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SUMMARY  AND  HIGHLIGHTS 


This  report  presents  the  results  of  an  analysis  of  the  manner  in 
which  information  is  utilized  from  the  Soil  Conservation  Service's 
Appraisal  Summary  (form  SCS-505)  in  order  to  determine  "Readiness  for 
Promotion"  to  Soil  Conservationist  (series  457)  GS-11  and  GS-12o 

The  major  categories  of  information  on  this  form  are  of  two  types, 
those  referring  to  an  individual's  characteristics  and  those  referring 
to  his  qualif icationSo 

It  was  found  that  there  was  little  or  no  gain  in  information  after 
the  six  most  informative  categories  were  considered  (see  Exhibit  1, 
page  6).  These  six  categories  were:  (1)  Leadership  qualities;  (2)  Date 
of  last  promotion;  (3)  Potential  at  two  GS  levels  higher;  (4)  Technical 
and  scientific  proficiency;  (5)  Organization  and;  (6)  Judgment. 

Some  assumptions  implicit  in  a Promotion  Appraisal  plan  as  well  as 
a means  for  improving  current  appraisal  procedures  through  performance 
documentation  are  discussed. 


Albert  S.  Glickman 
Chief, 

Personnel  Research  Staff 
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PROMOTION  TO  GS-11  AND  GS-12  SOIL  CONSERVATIONIST:  SOME 
EMPIRICAL  ANALYSES  OF  THE  DECISION-MAKING  PROCESS 


Introduction 


The  development  of  a career  plan  involves  the  design  of  a network 
of  choice  points  and  alternative  pathways  between  them.  At  each  choice 
point  there  is  constructed  a set  of  action  determinants»"organization 
policies,  personnel  needs,  laws,  regulations,  appraisals,  judgments, 
qualification  requirements  et  al“-“Which  control  or  influence  whether  an 
individual  will  pass  through  the  choice  point  and  the  path  or  pathways 
along  which  he  will  travel.  Thus,  a Soil  Conservationist  must  have 
certain  specified  qualifications  and  experiences  in  order  to  be  consi- 
dered as  a candidate  for  promotion  from  a GS'’!!  to  12,  Then,  a process 
of  judgments  are  applied  to  select  from  among  the  candidates  those  best 
fitted  to  -fill  the  position  or  positions  available.  Frequently,  quali- 
fications are  evaluated  and  a pool  of  eligible  individuals  ascertained 
before  the  openings  occur.  Then,  as  openings  occur  they  are  filled  from 
the  pool  of  eligibles. 

Regardless  of  whether  these  determinants  are  explicit  and  formally 
stated  or  are  implicit  and  informal,  they  function  operationally  to 
define  the  decision-making  process  that  must  be  performed  to  select 
individuals  and  to  meet  the  requirements  of  the  career  system. 

Reported  here  is  an  analysis  of  the  decision  process  currently 
involved  in  appraising  the  qualifications  of  candidates  for  promotion 
from  GS-9  to  GS-11  and  from  GS-11  to  GS-12  in  the  Soil  Conservationist 
(457)  series. 

The  information  used  in  this  analysis  was  taken  from  the  Soil  Con- 
servation Servicers  Appraisal  Summary  (form  SCS-505)  files.  Separate 
files  are  maintained  for  GS-9s  and  GS-lls.  A 50  percent  sample  of  each 
was  taken  by  pulling  every  other  appraisal  summary  from  the  file.  The 
information  on  these  forms  are  of  two  different  kinds;  (1)  those  per- 
taining to  the  individual’s  qualifications  (e.g.  education,  special 
training,  experience)  and  (2)  those  pertaining  to  his  characteristics 
(e.g.  leadership  qualities,  initiative,  ere,). 

Qualifications  data  are  derived  from  objective  information  in  per- 
sonnel folders. 

Characteristics  data  are  derived  from  supervisors*  judgments. 
Periodically,  each  individual  is  rated  on  each  of  16  characteristics  by 
two  to  four  (usually  three)  superiors.  At  the  same  time,  his  raters 
indicate;  (1)  whether  or  not  he  is  considered  "Ready  for  Promotion," 

(2)  what  kinds  of  positions  are  considered  most  suitable  for  him,  and 

(3)  an  estimate  of  the  grade  level  for  which  he  has  potential  to  rise 
in  the  long  run. 
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The  analysis  reported  here  focuses  on  the  contribution  of  these 
qualifications  and  characteristics  categories  to  an  individual's 
"Readiness  for  Promotion." 

In  order  to  subject  information  in  these  categories  to  a "multiple 
regression  analysis"  it  was  first  necessary  to  quantify  the  information 
that  appears  on  these  forms.  A description  of  this  coding  scheme  is 
given  in  Appendix  A. 

The  Utilization  of  Information  in  Multiple  Regression  Analysis 

The  manner  in  which  this  mathematical  technique  utilizes  information 
is  best  illustrated  by  a few  simple  diagrams. 

Suppose  that  we  have  two  categories  of  information  that  are  of 
interest  to  us.  These  can  be  represented  by  means  of  two  overlapping 
circles  or  sets  labeled  A and  B.  The  area  of  overlap,  the  shaded  area, 
represents  that  information  which  is  common  to  both  A and  B. 


Suppose  we  wanted  to  add  up  everything  in  A and  B by  adding  first 
everything  in  A and  then  everything  in  B.  In  doing  this  we  would  have 
counted  everything  in  the  shaded  area  twice,  once  when  counting  A and 
once  when  counting  B.  In  order  to  be  correct  in  our  arithmetic  we  only 
want  to  count  the  shaded  area  once,  hence  we  would  subtract  it  once 
(A+B  = A+  B-  ©))  . 

Now  let  us  complicate  the  picture  a little  by  introducing  a third 
set  C, 


and  let  us  define  the  following  areas; 

that  information  in  the  intersection  of  all  three  sets; 
that  information  common  to  B and  C which  A does  not  have; 
that  information  common  to  A and  C which  B does  not  have; 
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- that  information  common  to  A and  B which  C does  not  have. 


If  we  wanted  to  add  up  everything  in  all  three  sets,  A + B + C,  we 
would  end  up  counting  the  dotted  and  slanted  areas  twice  each  and  the 
solid  area  three  times.  To  correct  for  this  we  would  subtract  the 
dotted  and  slanted  areas  once  each  and  the  solid  area  twice,  i.e. 


A + B + C 


We  could  make  the  picture  increasingly  complex  by  adding  more  sets 
of  information,  however,  this  example  is  sufficient.  Let  us  ascribe 
some  meaning  to  these  categories  of  information. 

Let  us  now  assume  that  A,  B,  and  C are  three  measures  on  a group 
of  people: 

A - Score  for  each  person  representing  his  qualifications; 

B “ Score  for  each  person  representing  his  characteristics; 

C - Score  for  each  person  indicating  his  ’’Readiness  for  Promotion,” 

In  this  context  multiple  regression  analysis  would  attempt  to 
ascertain  the  manner  in  which  categories  A and  B are  combined  to  arrive 
at  an  individual’s  ’’Readiness  for  Promotion”  score.  The  outcome  would 
be  a set  of  weights  assigned  to  each  category  of  information  so  that 
when  combined  they  provide  the  ’’Readiness  for  Promotion”  score.  The 
weights  are  assigned  to  these  categories  so  as  to  maximize  their  rela- 
tionship with  the  ’’Readiness  for  Promotion”  score  and  minimize  the  amount 
of  overlap  between  categories  A and  B.  Thus,  in  the  following  example: 


The  solid  area  would  be  counted  as  part  of  A and  then  in  assigning 
a weight  to  B the  solid  area  would  not  be  counted  again.  Thus,  B would 
play  a much  smaller  part  in  determining  C than  would  A. 

An  individual’s  A score  would  then  be  multiplied  by  the  weight  for 
A,  his  B score  by  the  weight  for  B,  and  then  these  two  weighted  scores 
would  be  summed  in  order  to  arrive  at  an  estimated  C score.  A measure 
of  the  adequacy  with  which  the  C scores  are  estimated  by  this  weighting 
system  is  called  the  multiple  correlation.  This  value  varies  from  1.00 
for  perfect  estimation  to  .00  for  lack  of  any  relationship. 
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The  kinds  of  questions  one  seeks  to  answer  from  an  analysis  of  this 
nature  are: 

1.  Are  there  a few  crucial  categories  of  information  that  contri- 
bute most  to  the  overall  determination  of  "Readiness  for  Promotion?" 

2.  Do  promotion  policies  need  to  be  made  more  explicit  and  adminis- 
tered more  consistently? 

3.  Do  the  present  practices,  as  they  are  empirically  manifested  by 
the  analysis  of  the  current  decision-making  process,  result  in  the  selec- 
tion of  individuals  who  perform  well  in  the  positions  to  which  they  are 
promoted? 

Results  and  Discussion 

The  results  of  the  statistical  analyses  accompanied  by  a computational 
rationale  are  given  in  Appendix  B.  These  relate  to  a sequence  of  empirical 
questions  that  this  analysis  sought  to  answer  before  ascertaining  which 
were  the  crucial  categories  of  information. 

The  first  question  is:  Do  supervisors  tend  to  use  the  same  categories 
of  information  in  the  same  manner  in  determining  "Readiness  for  Promotion" 
to  GS-11  as  they  do  in  determining  "Readiness  for  Promotion"  to  GS-12? 

For  all  practical  purposes,  the  answer  is  "Yes."  (See  Appendix  B for 
detailed  analyses.) 

The  next  question  deals  with  the  contribution  of  two  characteristics 
that  were  added  to  the  appraisal  form  about  1961--Dependability  and  Over- 
all Performance.  Most  individuals  in  our  sample  were  evaluated  on  forms 
that  contained  these  two  characteristics.  Did  these  two  categories  add 
any  information?  That  is,  do  we  arrive  at  a different  determination  of 
"Readiness  for  Promotion,"  when  we  include  the  ratings  of  these  charac- 
teristics in  the  analysis  than  we  do  when  we  leave  them  out? 

The  difference  is  negligible;  therefore  we  conclude  that  the  cate- 
gories of  Dependability  and  Overall  Performance  did  not  add  any  new 
information--did  not  change  the  determination  of  "Readiness  for  Promotion" 
that  would  have  been  made  with  the  earlier  form  (see  Appendix  B) . 

The  analyses  up  to  this  point  have  shown  that:  (1)  the  same  cate- 
gories of  information  are  utilized  by  superiors  in  the  same  manner  in 
determining  "Readiness  for  Promotion"  for  GS-9s  and  GS-lls  and;  (2)  the 
addition  of  the  categories  of  Dependability  and  Overall  Performance  did 
not  contribute  any  new  information.  One  can  now  focus,  for  both  9s  and 
11s  combined,  on  what  categories  make  a crucial  contribution  to  the 
"Readiness  for  Promotion"  determination.  The  results  are  presented  in 
Exhibit  1 and  the  category  titles  are  listed  opposite  the  exhibit. 


Category  Titles 
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1,  Date  of  birth 

2.  Date  of  last  promotion 
So  Number  of  years  of  college  completed 

4.  Special  training  received 

5.  Number  of  merit  awards  within  the  last  five  years 

6.  Number  of  outstanding  performance  and/or  cash  awards 
five  years 

7.  Number  of  outstanding  performance  ratings  within  the 

8.  Number  of  other  kinds  of  awards  within  the  last  five 

9.  Technical  and  scientific  proficiency 

10.  Leadership  qualities 

11.  Production 

12.  Initiative 

13.  Judgment 

14.  Decision 

15.  Organization 

16.  Adaptability 

17 . Tact 

18.  Oral  communication 

19.  Written  communication 

20.  Working  with  others 

21.  Supervision 

22.  Using  instructions 

23.  Breadth  of  background 

24.  Number  of  positions  for  which  the  individual  has  potential  at  the 
next  highest  GS  level 

25.  Number  of  positions  for  which  the  individual  has  potential  at  two 
GS  levels  higher 

26.  Number  of  positions  for  which  the  individual  has  potential  at  three 
or  more  GS  levels  higher 


within  the  last 

last  five  years 
years 


If 

r • 


' 

1 

n 


. H 


I i 


t 


I,  i 


' l 
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CATEGORY  NUMBER  ADDED 

MULTIPLE  CORP.ELATION  .69.  HUMBER  OF  CANDIDATES  1,548.  * THE  SQUARED  MULTIPLE  CORRELATION. 
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In  light  of  these  results  we  can  answer  the  first  question;  Are 
there  a few  crucial  categories  of  information  that  contribute  most  to 
the  overall  determination  of  "Readiness  for  Promotion?" 

Inspection  of  Exhibit  1 shows  that  there  is  little  or  no  gain  in 
estimating  "Readiness  for  Promotion"  after  the  first  six  categories  are 
taken  into  account o The  rank  order  contribution  of  these  is  as  follows; 

1 Leadership  qualities  (10) 

2 Date  of  last  promotion  (2) 

3 Potential  at  two  GS  levels  higher  (25) 

4 Technical  and  scientific  proficiency  (9) 

5 Organization  (15) 

6 Judgment  (13) 

The  second  question  to  be  answered  is;  Do  policies  for  evaluating 
these  categories  need  to  be  made  more  explicit  and  administered  more 
consistent ly? 

When  the  multiple  correlation  is  high  it  indicates  considerable 
agreement  among  supervisors  in  the  categories  of  information  they  utilize 
and  the  weights  they  give  the  categories  in  arriving  at  a "Readiness  for 
Promotion"  score.  When  the  multiple  correlation  is  low  it  indicates  that 
supervisors  do  not  utilize  the  available  information  in  the  same  way. 

The  multiple  correlation  shown  at  the  bottom  of  Exhibit  1 is  ,69, 
indicating  that  agreement  among  raters  is  about  49%  better  than  would  be 
obtained  by  a random  assignment  of  ratings,  and  that  there  is  room  for 
improvement  to  be  made  in  the  reliability  with  which  supervisors  utilize 
the  information  in  these  categories  to  arrive  at  a "Readiness  for  Pro- 
motion" score.  Some  recommendations  as  to  how  greater  consistency  can 
be  achieved  are  taken  up  in  answering  the  final  question. 

The  final  question  is;  Do  the  present  practices,  as  they  are  mani- 
fested by  the  analysis  of  the  current  decision-making  process,  result  in 
the  selection  of  individuals  who  perform  well  in  the  positions  to  which 
they  are  promoted? 

This  question  can  be  answered  either  by  judgment  or  by  research. 

In  the  research  approach,  individuals'  information  category  "scores" 
on  the  Promotion  Appraisal  Summary  would  be  compared  against  their  per- 
formance on  the  jobs  to  which  they  had  been  promoted.  Those  categories 
that  best  predict  performance  at  the  next  higher  level  could  then  be 
identified.  A statement  as  to  the  success  of  the  promotion  evaluation 
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plan  could  then  be  made.  A high  relationship  between  certain  of  these 
categories  and  how  well  people  perform  at  the  next  higher  level  would 
indicate  that  the  program  is  successful“-that  superiors  had  a high 
"batting  average"  in  using  the  information  available  to  predict  degree 
of  success  of  subordinates. 

If  management  is  satisfied,  at  present,  with  the  performance  of 
people  being  promoted  under  this  system,  then  the  present  analysis 
shows  the  kinds  of  information  that  are  critical  in  making  these  deci- 
sions, However,  if  improvement  in  the  system  is  desired,  then  manage- 
ment might  scrutinize  at  least  some  of  the  assumptions  implicit  in  any 
Promotion  Appraisal  plan.  These  assumptions  are: 

(1)  that  the  supervisor  has  the  opportunity  to  observe  behaviors 
of  his  subordinates  which  are  indicative  of  how  well  they  will  perform 
at  the  next  higher  level  for  which  they  are  being  evaluated; 

(2)  that  the  supervisor  does  make  these  observations  and  records 
them;  and 

(3)  that  this  record  is  then  a basis  for  the  promotion  recommen- 
dations that  the  supervisor  makes. 

To  improve  the  system  management  would  want  to: 

(a)  find  out  what  behaviors  are  predictive  of  performance  at 
the  next  higher  level; 

(b)  insure  that  supervisors  know  what  behaviors  these  are;  and 

(c)  insure  that  these  behaviors  are  observed  and  recorded  as 
part  of  the  ongoing  supervisor --supervisee  relationship. 

Such  a program  is  currently  underway  with  the  Soil  Conservation 
Service  in  their  Critical  Incident  Survey,  This  survey  involves  the 
collection,  from  people  on  the  job,  of  examples  of  effective  and  inef- 
fective job  behaviors. 

In  the  creation  of  an  operational  system  the  remaining  steps 
involve: 

(1)  incidents  are  grouped  into  a set  of  logical  categories  of 
critical  job  requirements  which  are  presented  in  a Performance  Record 
form; 


(2)  supervisors  are  trained  to  observe  and  then  categorize  and 
record  these  incidents  in  the  Performance  Record;  and 

(3)  when  an  administrative  decision,  such  as  a promotion  recom- 
mendation, is  to  be  made,  the  supervisor  can  refer  to  his  documentations 
in  the  Performance  Record, 
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One  of  the  bases  for  a promotion  recommendation  would  be  the  extent 
to  which  the  same  kinds  of  critical  requirements  exist  at  both  the  level 
at  which  the  individual  is  currently  an  incumbent  and  the  level  for  which 
he  is  a candidate  for  promotion.  Thus,  if  an  individual  manifests  effec- 
tive behaviors  in  his  current  position  and  these  same  kinds  of  behaviors 
are  critical  requirements  at  the  next  higher  position,  he  would  be  an 
attractive  candidate  for  promotion. 


APPENDIX  A 
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SCS  Coding  for  Statistical  Analysis  of  Appraisal  Summary  GS  457 


Promotion  eligibility  for:  9 to  11,  457  series 

11  to  12,  457  series 

1.  Date  of  birth  - Use  the  last  three  digits  of  the  year  of  birth,  i.e., 
8_  _ or  9_ 

2.  Date  of  last  promotion  - Use  the  last  two  digits  of  the  year  of 
promotion,  i.e», 

3.  Education  - Years  of  college  completed 


High  school  graduate  or  less  00 

One  year  college  01 

Two  years  college  02 

Three  years  college  03 

Four  years  (Bachelor^s  degree  perhaps)  04 

Five  years  (Master's  degree  perhaps)  05 

Six  years  (Master's  degree  perhaps)  06 

Seven  years  (Ph.D.  degree  perhaps)  07 

Eight  years  (Ph.Do  degree  perhaps)  08 

Nine  years  09  etc. 


Count  each  additional  Bachelor's  degree  as  an  additional  year  of 
college , 

Count  each  additional  Master's  degree  as  an  additional  year  of  college. 

4.  Special  training  =>  Number  of  training  courses  or  schools  listed 
(00  to  _ _). 

5.  Number  of  merit  awards,  within  last  5 years  (0  to  9). 

6.  Number  of  outstanding  performance  awards  and/or  cash  awards,  within 
last  5 years  (0  to  9) . 

7.  Number  of  outstanding  performance  ratings,  within  last  5 years 
(0  to  9) o 

8.  Number  of  other  kinds  of  awards,  within  last  5 years  (0  to  9). 

9.  TOS 

10.  LQ 

11.  Production 


12. 


Initiative 


11 
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13.  Judgment 

14.  Decision 

15.  Organization 

16.  Adaptability 

17.  Tact 

18.  Oral  communications 

19.  Written  communications 

20.  Working  with  others 

21.  Supervision 

22.  Using  instructions 

23.  Dependability 

24.  Overall  performance 

To  score  9 through  24  assign  values  of  2 to  strong,  1 to  average, 
and  0 to  weak.  Multiply  these  by  the  number  of  X's  in  category  for  each 
characteristic,  add  these  up  and  divide  by  the  total  number  of  X's  for 
that  characteristic.  For  exan^le: 


Average  Weak 

1 0 

XX  X 

2 0 = 4=1 

4 

This  score  will  range  from  0 to  2,  i.e.,  £ to  ^ (multiply  each  score 

4 4 

by  100  and  retain  the  first  three  numbers). 

25.  Breadth  of  background  - The  total  number  of  positions  to  which  an 
individual  could  be  reassigned  (0  to  . 

26.  No.  of  positions  for  which  the  individual  has  potential  (0  to  _)  at 
the  next  highest  GS  level. 

27.  No.  of  positions  for  which  the  individual  has  potential  (0  to  _)  at 
two  GS  levels  higher. 


Strong 

2 

9.  Tech.  & Scien.  Proficiency  X 

2 


?'  I 


I 

I 

i 


''-t 
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28 o No.  of  positions  for  which  the  individual  has  potential  (0  to  __)  at 
three  or  more  GS  levels  higher. 

29.  Average  number  of  raters  recommending  promotion  to  Work  Unit  Conser- 
vationist - Divide  the  number  of  X' s indicating  readiness  for  promo- 
tion to  Work  Unit  Conservationist  by  the  total  number  of  raters. 

This  score  will  range  from  0 to  1,  i.e.  0 to  4 (multiply  each  score 

4 4 

by  100  and  retain  the  first  three  numbers) . 
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Computational  Rationale  for  the  Sequence  of  Empirical  Questions 


A full  explaiiation  of  the  following  procedures  is  given  in 
Bottenberg,  R.  A,,  and  Ward,  J,  H„ , Jio,  Applied  Multiple  Linear  Regres- 
sion, Technical  Documentary  Report  PRL-TOR"63-6,  March  1963,  6570th 
Personnel  Research  Laboratory,  Aerospace  Medical  Division,  Air  Force 
Systems  Command,  Lackland  Air  Force  Base,  Texas. 

1.  Can  the  same  regression  equation  be  used  to  predict  "Readiness 
for  Promotion"  from  GS"9  to  11  as  from  GS-11  to  12?  This  question  is 
answered  by  observing  the  magnitude  of  the  difference  between  the 
following  two  squared  multiple  correlations  (SMC’s):  (1)  the  SMC  com- 
puted on  all  26  categories  (Overall  Performance  and  Dependability 
omitted)  for  both  groups  combined;  (2)  the  SMC  computed  on  all  26  cate- 
gories and  a 27th  category  introduced  for  group  membership  (scored  1 
for  a 9 and  0 for  an  11).  One  way  to  evaluate  the  magnitude  of  this 
difference  (given  in  Exhibit  2)  is  by  reference  to  the  F statistic  using 
the  appropriate  degrees  of  freedom  (see  Bottenberg  and  Ward,  P.  125-126). 
The  differences  obtained  in  this  analysis  were  evaluated  against  the 
criterion  of  loss  of  predictive  power  versus  ease  of  using  a single 
regression  equation.  With  this  criterion  a difference  of  ,04  reported 

in  Exhibit  2 was  judged  not  to  be  appreciable  enough  to  depart  from  a 
single  regression  equation. 

2,  Do  the  categories  of  Dependability  and  Overall  Performance 
contribute  any  information  in  addition  to  that  in  the  other  categories? 
This  question  is  answered  by  observing  the  magnitude  of  the  difference 
between  the  following  two  SMC's  computed  on  both  9s  and  11s  who  have 
these  two  additional  categories”  (1)  the  SMC  computed  when  these  two 
categories  are  excluded  from  the  analysis  and;  (2)  the  SMC  computed 
when  these  two  additional  categories  are  included  in  the  analysis. 

The  difference  of  .01  given  in  Exhibit  3 is  not  appreciable  enough  to 
consider  them  as  contributing  any  differential  information. 


Exhibit  2 

Squared  Multiple  Correlations  When  GS"9  and  GS-11 
Differences  Are  and  Are  Not  Taken  into  Account 


Taken  into  account 

SMC  = 

.52 

Not  taken  into  account 

SMC  = 

00 

Difference 

,04 

I 
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Exhibit  3 

Squared  Multiple  Correlations  When  Dependability  and 
Overall  Performance  Are  Included  and  Excluded 


Included 

SMC  = 

.54 

Excluded 

SMC  = 

.53 

Difference 

.01 
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